Evaluation of machine translation by reading comprehension tests and subjective judgments

This paper discusses the results of an experiment designed to test the quality of translations, in which human subjects were presented with IBM-produced machine translations of several passages taken from the Russian electrical engineering journal Elektrosviaz, and with human translations of some other passages taken from Telecommunications, the English translation of Elektrosviaz. The subjects were tested for comprehension of the passages read, and were also asked to judge the clarity of individual sentences. Although the human translations generally gave better results than the machine translations, the differences were frequently not significant. Most subjects regarded the machine translations as comprehensible and clear enough to indicate whether a more polished human translation was desirable. The reading comprehension test and the judgment of clarity test were found to give more consistent results than an earlier procedure for evaluating translations, since the questions asked in the current series of tests were more precise and limited in scope than those in the earlier scries.